Models as Agents: Optimizing Multi-Step Predictions of Interactive Local Models in Model-Based Multi-Agent Reinforcement Learning

Research in model-based reinforcement learning has made significant progress recent years. Compared to single-agent settings, the exponential dimension growth of joint state-action space multi-agent systems dramatically increases complexity environment dynamics, which makes it infeasible learn an accurate global model and thus necessitates use agent-wise local models. However, during multi-step rollouts, prediction one can affect predictions other models next step. As a result, errors be propagated localities eventually give rise considerably large errors. Furthermore, since are generally used predict for multiple steps, simply minimizing one-step regardless their long-term effect on may further aggravate propagation To this end, we propose Models as AGents (MAG), optimization framework that reversely treats decision making agents current policies dynamics rollout process. In way, able consider mutual between each before predictions. Theoretically, show objective MAG is approximately equivalent maximizing lower bound true return. Experiments challenging StarCraft II benchmark demonstrate effectiveness MAG.